Search supporting system, search supporting method and search supporting program

ABSTRACT

In a database, product image data is accumulated. A search portion acquires product image data having the image characteristics information that is the same as or similar to the image characteristics information that indicates the characteristics of the image of input image data from the database for the input image data. A search server outputs information on another product that is different from the product corresponding to the product image data together with the product image data acquired by the search portion.

This application is a divisional of U.S. patent application Ser. No.13/629,338 filed Sep. 27, 2012, which in turn is a divisional of U.S.patent application Ser. No. 12/461,328 filed Aug. 7, 2009, is anon-provisional application claiming priority to and the benefit of U.S.provisional application No. 61/136,293, filed Aug. 26, 2008, and claimspriority to Japanese Patent Application No. 2008-205730 filed on Aug. 8,2008 and Japanese Patent Application No. 2008-262035 filed on Oct. 8,2008. The entire contents of which are incorporated herein by reference.

BACKGROUND Field of the Invention

The present invention relates to a search supporting system, a searchsupporting method, and a search supporting program that support a userto search a target through the Internet and to select or determine atarget.

Related Art

Conventionally, systems that support selection or determination of atarget to be searched by a user have been used when the user searchesfor the target (for example, a product or the like) through a networksuch as the Internet and purchases the target (for example, see JapanesePatent Application Publication No. 2004-246585A).

For example, a user searches for a target by inputting the name (forexample, a product name or the like) of the target or thecharacteristics of the target by using various search engines availableon the Internet. In addition, as needed, the user searches for a storethat sells the above-described target.

Then, a server device having the above-described search engine supportsthe user's search for the target by presenting the image of a targetcorresponding to the user's preference or the price thereof included inthe found store to the user's terminal through the Internet or providinga related target or the like.

In addition, recently, the purchase of products from a virtual storefrom a virtual mall on the Internet without visiting a retail store thatprovides face-to-face sales for purchasing the product has increased.

As a result, even in a region in which a large-size retail store is notlocated nearby, it becomes possible for a user to purchase a product ofchoice from among a plurality of products of many types while beingseated in his or her house (for example, see Japanese Patent ApplicationPublication No. 2002-150138A).

However, in the above-described conventional examples and the like, whenthe user searches for the target, the user inputs text data representingthe name or the like of the target or selects the target from images oftargets that have been prepared in advance for a search. Accordingly,when the user has information on the image of the target only and doesnot know the name of the target, there is a problem that the user cannotsearch the target actually desired to be searched or a target that issimilar to the search target.

A purpose of some aspects of the present invention is to provide asearch supporting system, a search supporting method, and a searchsupporting program that are capable of searching for a target desired tobe searched by the user by using the image data of the target, forexample, even in a case where the name of the target is not known to theuser.

SUMMARY

According to an aspect of the invention, a search supporting system isprovided, the system including: a database in which product image datais accumulated; and a search portion that acquires the product imagedata having image characteristics information that is the same as orsimilar to the image characteristics information representing thecharacteristics of an image of input image data for the input image datafrom the database, wherein information on another product that isdifferent from a product corresponding to product image data is outputtogether with the product image data that is acquired by the searchportion.

According to another aspect of the invention, a method of supporting asearch is provided, the method including: a search process for acquiringproduct image data having image characteristics information that is thesame as or similar to the image characteristics information thatrepresents the characteristics of an image of input image data for theinput image data from a database; and a process for outputtinginformation on another product that is different from a productcorresponding to the product image data together with the product imagedata that is acquired by the search process.

According to another aspect of the invention, there is provided a searchsupporting program that is a computer-executable search supportingprogram that allows a computer to perform the operations of: a searchprocess for acquiring product image data having image characteristicsinformation that is the same as or similar to the image characteristicsinformation that represents the characteristics of an image of inputimage data for the input image data from a database; and a process foroutputting information on another product that is different from aproduct corresponding to the product image data together with theproduct image data that is acquired by the search process.

According to another aspect of the invention, a search supporting systemis provided, the system including: a database in which target image datafor a target searched by a user is accumulated; an extraction portionthat extracts a searching portion of input image data that has beeninput; a search portion that acquires the target image data thatcoincides with or has high similarity to image data of the searchingportion from the target image data from the database by comparing imagecharacteristics information that represents the characteristics of animage of the image data of the searching portion with imagecharacteristics information that represents the characteristics of animage of the target image data that is included in the database.

According to another aspect of the invention, a method of supporting asearch is provided, the method including: an extraction process forextracting a searching portion of a target from input image data thathas been input; and a search process for extracting the target imagedata that coincides with or is highly similar to the image data withinthe searching portion by comparing the image data within the searchingportion and the target image data that is accumulated in the databasefrom the database.

According to another aspect of the invention, there is provided a searchsupporting program that is a computer-executable search supportingprogram that allows a computer to perform the operations of: anextraction process for extracting a searching portion of a target frominput image data that has been input; and a search process forextracting the target image data that coincides with or is highlysimilar to the image data within the searching portion by comparing theimage data within the searching portion and the target image data thatis accumulated in the database from the database.

According to another aspect of the invention, a search supporting systemis provided, the system including: a combined product database in whichattribute information of product items acquired form an image medium isaccumulated for each same category; a combination information databasein which, for each of the product items, combination information withanother product item of a different category that is used in combinationwith the each of the product items is stored; a product item database inwhich attribute information of product items that are sold is stored; asimilar-item searching portion that selects the image data of a productinput by a user and a candidate group of the product items of whichattribute information is the same as or similar to that of the imagedata from the combined product database; a combination searching portionthat, in correspondence with each of the product items of the candidategroup, searches for other product items that are combined with the eachof the product items from the combination information database; and aproduct-item searching portion that searches for image data of a productitem that is the same as or similar to the image data of the anotherproduct item of the combination that is selected by the user fromcombinations of the product item and the another product item from theproduct item database based on the attribute information of the imagedata and outputs the image data of the another product item as arecommended product.

According to another aspect of the invention, a method of supporting aproduct search is provided, the method including: a similar-itemsearching process for selecting a candidate group of product items thatare the same as or similar to image data of a product input by a userand attribute information of the image data from a combined productdatabase in which the attribute information of the product itemsacquired from an image medium are accumulated for each same category; acombination searching process for searching for another product itemthat is combined with the product item in association with each productitem of the candidate group from a combination information database inwhich combination information between each product item and anotherproduct item of a different category that is used in combination withthe each product item is stored for the each product item; and a productitem searching process for searching for the product item that is thesame as or similar to the image data of the another product item of thecombination that is selected by the user from the combinations betweenthe product item and the another product item based on the attributeinformation of the image data of the another product item and the imagedata from a product item database in which the attribute information ofthe product items is accumulated and outputs the product item as arecommend product.

According to another aspect of the invention, there is provided a searchsupporting program that is a computer-executable search supportingprogram that allows a computer to perform the operations of: asimilar-item searching process for selecting a candidate group ofproduct items that are the same as or similar to image data of a productinput by a user and attribute information of the image data from acombined product database in which the attribute information of theproduct items acquired from an image medium are accumulated for eachsame category; a combination searching process for searching for anotherproduct item that is combined with the product item in association witheach product item of the candidate group from a combination informationdatabase in which combination information between each product item andanother product item of a different category that is used in combinationwith the each product item is stored for the each product item; and aproduct item searching process for searching for the product item thatis the same as or similar to the image data of the another product itemof the combination that is selected by the user from the combinationsbetween the product item and the another product item based on theattribute information of the image data of the another product item andthe image data from a product item database in which the attributeinformation of the product items is accumulated and outputs the productitem as a recommend product.

According to some aspects of the present invention, a target desired tobe searched by a user or a target similar to the above-described targetcan be searched in an easy manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram representing an example of the configurationof a search supporting system according to an embodiment of the presentinvention.

FIG. 2 is a conceptual diagram representing an example of theconfiguration of a product table that is stored in a database 16represented in FIG. 1.

FIG. 3 is a conceptual diagram representing an example of theconfiguration a user registration table that is stored in the database16 represented in FIG. 1.

FIG. 4 is a flowchart representing an example of the operation ofproduct searching in a search supporting system according to theembodiment.

FIG. 5 is a flowchart representing an example of the operation ofpreference information collecting in a search supporting systemaccording to the embodiment.

FIG. 6 is a block diagram representing a modified example of the exampleof the configuration of the search supporting system according to thisembodiment.

FIG. 7 is a block diagram representing an example of the configurationof a product search supporting system according to an embodiment of thepresent invention.

FIG. 8 is a table in which attribute information of a combinationproduct item of Category 1 (shirts) in a product item database 26represented in FIG. 7.

FIG. 9 is a table in which attribute information of a combinationproduct item of Category 2 (pants) in the product item database 26represented in FIG. 7.

FIG. 10 is a table in which attribute information of a combinationproduct item of Category 3 (jacket) in the product item database 26represented in FIG. 7.

FIG. 11 is a table representing correspondence between a product item ofCategory 1 that is sold and a combined product item of Category 1 thatis collected from an image medium similar to the product item.

FIG. 12 is a table representing correspondence between a product item ofCategory 2 that is sold and a combined product item of Category 2 thatis collected from an image medium similar thereto.

FIG. 13 is a table representing correspondence between a product item ofCategory 3 that is sold and a combined product item of Category 3 thatis collected from an image medium similar thereto.

FIG. 14 is a table representing a combination between categories of thecombined product items collected form an image medium in a combinationinformation database 28 of FIG. 7.

FIG. 15 is a table in which attribute information of a combined productitem of Category 1 (shirt) of the combined product image informationdatabase 29 of FIG. 7.

FIG. 16 is a table in which attribute information of a combined productitem of Category 2 (pants) of the combined product image informationdatabase 29 of FIG. 7.

FIG. 17 is a table in which attribute information of a combined productitem of Category 3 (jacket) of the combined product image informationdatabase 29 of FIG. 7.

FIG. 18 is a flowchart representing an example of the operation of theproduct search supporting system of FIG. 7.

FIG. 19 is a conceptual diagram representing the configuration of a usertable of a history database 31 of FIG. 7.

FIG. 20 is a conceptual diagram representing the configuration of apurchase history table of the history database 31 of FIG. 7.

FIG. 21 is a conceptual diagram for describing division of image data byusing a Graph-Cut method.

DESCRIPTION OF EMBODIMENTS First Embodiment

Hereinafter, a search supporting system according to an embodiment ofthe present invention will be described with reference to drawings. FIG.1 is a block diagram representing an example of the configuration of thesearch supporting system according to the embodiment.

In the figure, the search supporting system includes a search server 1and a user terminal 2 connected to the search server 1 through aninformation communication network I configured by the Internet or thelike.

Here, the user terminal 2 is a terminal that is used by a user and isidentified by user identification information that is unique to eachuser.

Hereinafter, in this embodiment, a case where a user searches for aproduct such as a fashion product, clothes, shoes, a necklace, or a hatas a desired search target and purchases the product will be describedas an example. In addition, this embodiment is not limited to the caseof a product such as a fashion product, clothes, shoes, a necklace, or ahat and can be applied to the case of a target such as an electronicproduct, furniture, or a painting that can be searched for through theInternet or a network. Furthermore, examples of the target according tothis embodiment are a commercial product, goods, an electronic product,furniture, a painting, a building that includes a store or a company, aplant, an animal, real estate (includes the exterior, the interior, thearrangement of rooms, or the like of a mansion), a landscape, and thelike.

The search server 1 is a server that supports a user's search for aproduct in a store on the internet or the like and includes anextraction portion 11, a type determining portion 12, a search portion13, a combination extracting portion 14, a preference extracting portion15, a database 16, and a transmission portion 17.

In the database 16, as shown in FIG. 2, a product table in which, inassociation with product identification information for indentifyingeach product, the name of a product, product image data (target imagedata) that is image data of the product, characteristics informationthat is extracted from the product image data, store information inwhich the product is sold, the price of the product, combinationinformation of other products combined with the product, and typeidentification information that represents the type (a product genresuch as clothes, shoes, a hat, or a necklace) are associated with oneanother is stored. In this database 16, for example, product informationof stores that are registered in the search supporting system as membersis sequentially accumulated. The above-described characteristicsinformation is formed by representing elements such as a color, a form,a shape, and a texture as numeric data (for example a vector of theorder corresponding to the number of the elements). When images aredirectly compared to each other based on the image of a product desiredto be searched by the user and the product image (target image) that isaccumulated in the database 16, the type determining portion 12 may beomitted.

The extraction portion 11 extracts a search portion from the input imagedata by receiving image data (the image data provided by the user),which is transmitted from the user terminal 2 by the user, through thetransmission portion 17. Described in more details, the extractionportion 11 extracts the contour (search portion) of the image area of aproduct from the input image data and generates contour image data (theimage data of the search portion). Here, in the process of extractingthe contour image data, a contour is a portion in which the densityvalue of the image is abruptly changed. In order to extract the contour,a differential operation is performed. However, for a digital image,data is aligned at a constant interval in a discontinuous manner, andaccordingly, as an operation (difference) for taking a differencebetween adjacent pixels, the differential is approximated, and a portionin which the density of the pixel is abruptly changed is extracted as acontour.

The type determining portion 12 searches for template image datacorresponding to template image data from the template image data in theproduct contour data table that is stored in the database 16 in advanceby using the above-described contour image data and reads in the typeidentification information that is set in correspondence with thetemplate image data in the database 16. It is preferable that thistemplate image data be stored as templates to be compared with thecontour included in the contour data table of the database 16 in advanceby photographing a plurality of products for each type from a pluralityof angles.

The search portion 13 extracts the characteristics of the search imagedata acquired by performing contour extraction for the input image datafrom a memory area of the type corresponding to the above-described typeidentification information and searches for characteristics informationthat coincides with the acquired characteristics information or has highsimilarity to the acquired characteristics information from the imagedata that has the type identification information corresponding to thedetermined type in the product table of the database 16, and extractsthe image data of products having high similarity, the number of whichis a value set in advance, in the descending order of the similarity.Here, in a case where the type determining portion 12 is not provided,the search portion 13 compares the image of the product of which thecontour is extracted from the input image data with the product imagedata included in the whole product table and extracts product image datahaving high similarity to the image of the product for which the contourextraction has been performed.

In addition, the search portion 13 transmits the product information(target information) corresponding to the searching product image datato the user terminal 2 used by the user. In addition, theabove-described product information is information that includes atleast one of the name (product name), the sales store that sells theproduct corresponding to the product image data, the URL of the salesstore, the telephone number or the address of the sales store, and theprice of the product.

The preference extracting portion 15 extracts image data of otherproducts (products of the same type as that of the product searched bythe user) included in the type of the product that has been searched bythe user, the number of which is randomly set in advance, from theproduct table and transmits the image data to the user terminal 2.

The user terminal 2 displays the image data that is transmitted from thesearch server 1 on a display screen not shown in the figure.

In addition, the preference extracting portion 15 writes preferenceinformation, which includes at least one of the form, the color, theshape, and the texture for each user into a user registration table ofthe database 16 that is represented in FIG. 3 for record by inputtingthe result of determination on likes or dislikes for each product (forexample, the product image data stored in the database 16) that istransmitted from the user terminal 2.

Here, in the user registration table, purchase information (includingthe purchased product identification information, purchased date andtime, the store at which the product is purchased, the purchase price,and the like) that indicates products that were purchased in the past,the preference information that is extracted by the preferenceextracting portion 15, and point information that represents the numberof points owned by the user are stored in association with the useridentification information. These points can be used like cash at thetime of paying for a purchased product.

In other words, in a case where discount coupon information is added tothe above-described store information and the user purchases a foundproduct by using the discount coupon information, the search server 1stores the purchase information on the product in the above-describeduser registration table of the database 16. Then, the search server 1adds points corresponding to the price of the purchased product to thecorresponding user included in the user registration table as the numberof points of the above-described point information. The preferenceextracting portion 15 may be configured to extract the preferenceinformation for each type from the above-described purchase information.

The combination extracting portion 14 transmits products of other typesthat may be purchased in combination with the extracted product at highpossibility to the user's terminal by searching in advance for productscorresponding to the combination information that is stored inassociation with the preference information.

Next, the operation of the search supporting system according to thisembodiment will be described with reference to FIGS. 1 and 4. FIG. 4 isa flowchart for describing an example of the operation of the searchsupporting system according to this embodiment. In descriptions below, acase where the type determining portion 12 is provided will bedescribed. When the type determining portion 12 is not provided, theprocess of the type determining portion 12 is omitted from theflowchart.

A user reads out the image of a model wearing a favorite clothing from amagazine or the like by using a color scanner or the like that isconnected to the user terminal 2, performs photographing by using adigital camera, or downloads an image from the Internet (Step S1).

Then, the user inputs the read-out image, the photographed image, or thedownloaded image to the user terminal 2 as input image data anddetermines whether to select an area of the image of a product desiredto be searched for based on whether there is a plurality of types ofproducts in the input image data (Step S2). In addition, the input imagedata may be stored in the user terminal 2 in advance or may be stored inan external terminal.

At this moment, for example, when the product name of a skirt and astore selling the skirt are to be searched for, in a case where only theskirt shows up in the input image data, the process proceeds to Step S4.

On the other hand, when the entire model, not only the image of theskirt portion, is photographed, the process proceeds to Step S3.

Then, the user selects an image by marking the image of the skirt areaportion with a line or the like by using an image processing tool (anytool that can be used for performing a process of drawing a line in theinput image data) that is installed to the user terminal 2 (Step S3),and the process proceeds to Step S4.

The user terminal 2 transmits the input image data (including data forwhich image selection has been performed) input by the user togetherwith its user identification information to the search server 1 throughthe information communication network I as a search request signal (StepS4).

In addition, in the above-described Step S2, it may be configured thatthe search server 1 determines whether there is a plurality of types ofproducts in the input image data received from the user terminal 2 andtransmits the determined type to the user terminal 2. In this case, inStep S3, image selection is performed for the type desired to besearched for from among the types included in the input image datareceived from the search server 1, and the process proceeds to Step S4.

Next, when receiving the above-described search request signal from theuser terminal 2, the search server 1 outputs the above-described inputimage data to the extraction portion 11.

The extraction portion 11 performs contour extraction (extraction of thesearch portion) for the product image included in the input image datathat has been input.

At this moment, when any area included in the input image data is notmarked, the extraction portion 11 extracts contour image data of theentire input image data. On the other hand, when there is an areaportion that has been marked, the extraction portion 11 extracts thecontour image data of the image of the marked area portion.

Then, the type determining portion 12 searches for template image datacorresponding to the contour image data from the contour data tablecorresponding to the type of the product stored in the database 16 inadvance by using the above-described contour image data and reads in thetype identification information corresponding to the template image datafrom the contour data table (Step S5).

The search portion 13 extracts the characteristics of the image data(search image data) to be searched for, which is included in the inputimage data, in association with the contour inner portion of theabove-described contour image data and searches for the product imagedata similar to the characteristics information (for example,information on the basis of the color or the form) of the search imagedata from the product table of the database 16 (Step S6).

At this moment, the search portion 13 requests for vector distancesamong elements (for example, in a case where the characteristicsinformation is information on the basis of the color, blue, red, yellow,and the like) of the characteristics information and calculates thesimilarity indicating whether the distances coincide with each other orthe distance is short or long. It is assumed that the similarity is highfor a case where the distance is short.

In addition, the search portion 13 extracts products, the number ofwhich is a value set in advance, in the descending order of thesimilarity from the product table.

Next, the search portion 13 transmits product information (informationthat includes at least one of the product name, the price of theproduct, the store selling the product image data, the URL, thetelephone number and the address of the store) corresponding to thefound product image data to the user terminal 2 that is used by the userthrough the transmission portion 17 (Step S7). Here, as an example, thetransmission portion 17 reads out the address of the user terminal 2 onthe network, which is added at the time of transmission of the searchrequest signal from the user terminal 2, and transmits theabove-described product information to this address.

Then, the user terminal 2 displays the information on the product name,the image data, the store, and the price as the search result that hasbeen transmitted from the product sales searching server 1 on thedisplay screen.

In addition, in the above-described Step S5, in the user registrationtable of the database 16, the search server 1 may be configured toextract the product image data of the search target from the database 16based on the input image data that is included in the search requestsignal only in a case where registration of the user identificationinformation, which is included in the search request signal, in the userregistration table is detected. In addition, when detecting that theuser identification information included in the search request signal isnot registered in the user registration table, for example, the searchserver 1 transmits information to the user terminal 2 indicating thatthe user registration on the basis of the user identificationinformation is needed.

Accordingly, the user can acquire information on the product name of afavorite product or a product similar to the favorite product, storeinformation that indicates the store that sells the product, the priceof the product, and the like by transmitting the image data.

Here, in a case where discount coupon information is added to the storeinformation, and the found product is purchased by using the discountcoupon information, the product sales searching server 1 charges anamount corresponding to the price of the product to the target store.

Next, a process for extracting the user's preference information andsuggesting other products that can be combined together with theabove-described found product based on the preference information byusing the search server 1 according to this embodiment will be describedwith reference to FIGS. 1 and 5. FIG. 5 is a flowchart for describing anexample of the operation for introducing other products based on thepreference information by collecting the user's preference information.

The preference extracting portion 15 extracts products of the same typeas that of the user's searching product, for example, shoes for a casewhere the user's searching products are shoes, the number of which is avalue set in the product table in advance (Step S11), and transmits theproduct image data and the product identification information (StepS12). At this moment, the products for each type to be transmitted tothe user are odd products that are different from one another in thecolor, the form, the shape, and the like and are configured as productsfrom which a fashion coordinator can extract the user's preferenceinformation.

Then, when receiving the product image data of the products forextracting the preference from the search server 1, the user terminal 2sequentially displays the product image data (product images) of theproducts on the display screen.

At this moment, in the display image displayed on the above-describeddisplay screen, when the user likes the product of the displayed productimage data, the user clicks on the “positive button” by using inputmeans such as a mouse. On the other hand, when the user does not likethe product of the displayed product image data, the user clicks on the“negative button” by using the input means such as a mouse. (Step S13).

When the “positive button” or the “negative button” is selected, theuser terminal 2 displays the product image data of the next product onthe display screen, and this process is continued until the user'spositive or negative selection for all the received product image dataor the received product image data corresponding to a predeterminednumber is completed.

When the above-described positive or negative determination for theabove-described product image data is completed, the user terminal 2transmits the search server 1 the determination result information inwhich the positive or negative determination result for the product isassociated with each product identification information with its useridentification information added thereto (Step S14).

When receiving the determination result information as input, thetransmission unit 17 determines whether the added user identificationinformation is registered in the user registration table. When the addeduser identification information is registered in the user registrationtable, the transmission unit 17 outputs the above-describeddetermination result information to the preference extracting portion15.

Next, when receiving the determination result information as input, thepreference extracting portion 15 selects preference informationcorresponding to the positive and negative determination pattern fromthe preference information table of the database 16.

Here, the positive and negative determination pattern is a pattern ofpositive determination and negative determination corresponding to theproduct identification information that is arranged in the order set inadvance.

Then, the preference extracting portion 15 writes the acquiredpreference information in the user registration table of the database 16in association with the user identification information (Step S15).

Next, the combination extracting portion 14 selects combinationinformation of the products corresponding to the above-describedpreference information from the combination information table of thedatabase 16. In this combination information table, for example, in acase where the preference information is extracted with respect toshoes, combination information corresponding to the clothes, the bag,the hat, and the like corresponding to the preference is selected.

Then, the combination extracting portion 14 extracts the product imagedata of the clothes, the bag, the hat, and the like that coincide withor is similar to the above-described combination information (Step S16).Then, the combination extracting portion 14 transmits one type or aplurality types of wearing image data, in which a model wears theclothes, the bag, the hat, and the like that have been selected to matchwith the shoes, to the user terminal 2 by adding the combinationidentification information to each combination (Step S17).

When receiving the wearing image data, the user terminal 2 displays thewearing image data on the display screen. Accordingly, the user canacquire information on the combination of fashions corresponding to hisor her preference for the product that has been searched for orpurchased.

Here, by the user's performing the positive and negative determinationas described above, the user terminal 2 transmits the positive andnegative determination data in association with each combinationidentification information to the search server 1 as the determinationresult information with its user identification information addedthereto.

Then, when receiving the determination result information as input, thetransmission portion 17 determines whether the added user identificationinformation is registered in the user registration table. When the useridentification information is registered in the user registration table,the transmission portion 17 outputs the above-described determinationresult information to the preference extracting portion 15.

The preference extracting portion 15 generates combination preferenceinformation based on the positive and negative determination patterncorresponding to the input combination identification information. Thiscombination preference information is formed by a combination of theform, the color, the shape, and the texture, and the like of otherclothes, bags, hats, and the like that have been positively determinedin association with the form, the color, the shape, the texture, and thelike of the shoes. Every time the user purchases a product, thepreference is learned, whereby the accuracy of the combinationpreference information is improved.

In other words, when the user purchases a bag based on the combinedwearing image data, the preference information for the bag is searchedfor, and the combination preference information of the shoes, theclothes, the hat, and the like corresponding to this preferenceinformation can be acquired, whereby the preference of each user issequentially narrowed.

In addition, as a challenge combination (a combination that is notnormally selected by a user, or a combination that is different from theabove-described preference information), it may be configured that afashion coordinator randomly selects several products corresponding tocombination information, which is extremely different from thecombination information selected based on the preference information,from the product table, and theses selected products are inserted intoseveral combinations at the time of generation of the wearing imagedata.

Accordingly, the user's preference is diversified, whereby there is apossibility that the user's purchase desire increases.

In addition, the search supporting system according to this embodiment,as shown in FIG. 6, the user terminal 2 may include a user terminal 2 aconfigured to have an extraction portion 11 a that has the sameextraction function as that of the above-described extraction portion 11and a search server 1 a other than the extraction portion 11. In such acase, the extraction portion 11 a extracts a search portion (forexample, a product portion) of input image data based on the input imagedata acquired by the user terminal 2 a or the input image data stored inthe user terminal 2 a and transmits only the image data corresponding tothe search portion to the search server 1 a. In addition, since portionsother than the extraction portion 11 a are the same as those of theabove-described embodiment, a description thereof is omitted. Asdescribed above, the user may transmit only the image data correspondingto the search portion of the input image data to the search server 1 aas the image data to be transmitted. For example, the image datacorresponding to the user's face portion or other persons' face portionsneeds not to be transmitted to the search server 1 a.

In addition, in this embodiment, the search supporting system of thepresent invention may be applied not only to the products such asclothes or shoes but also to the above-described targets. In addition,for example, the search supporting system of the present invention maybe applied to services that are provided in an accommodation facility (ahotel or an inn), a restaurant, or the like. For example, in theabove-described accommodation facility, the search supporting system ofthe present invention may be applied to a case where a combination ofroom arrangements, a combination of sceneries of windows, or acombination of rooms or room layouts and sceneries of windows, or thelike is selected. In addition, for example, in the above-describedrestaurant or the like, the search supporting system of the presentinvention may be applied to a case where a combination of interiors ofthe restaurant, a combination of sceneries of windows, a combination ofmusic, a combination of an interior and the scenery of a window, acombination of an interior and music, a combination of the scenery of awindow and music, or the like is selected.

Second Embodiment

Hereinafter, a product search supporting system according to anembodiment of the present invention will be described with reference todrawings.

Conventionally, when combined products (for example, garments, combinedfurniture, combined electric appliances, or the like) acquired bycombining a plurality of products of different types (categories) are tobe purchased, in a case where a customer does not actually go to aretail store, the customer cannot directly ask store personnel whichcombination is appropriate.

In addition, when a user selects a product, which matches well a productpurchased by the user or a product to be purchased for a case where theproducts are combined, for himself from a vast product group of avirtual store, the user cannot check one by one with a coordinator ofthe combined products, and the user cannot select the product forhimself. Accordingly, the user cannot purchase the product.

For example, even when the user wants to select pants or a shirt thatmatch a jacket purchased by the user from garments, the user cannotcheck the selection with a first-class fashion coordinator and worriesabout selection of an inappropriate combination, and accordingly, theuser gives up the purchasing of matching pants or a matching shirt.

Here, the garments represent all the clothes and personal ornaments (anaccessory, a bag, shoes, a hat, and the like) that are worn around anatural human body in the same state as it is born.

This embodiment also solves such a problem. This embodiment supports auser's purchase of combined products by extracting products that match aproduct that is to be purchased or has been purchased by the user andrecommending the extracted products to the user in purchasing thecombined products.

FIG. 7 is a block diagram representing an example of the configurationof the product search supporting system according to this embodiment.The invention is a system that supports a user to easily select aproduct item, which is thought to form an excellent combination with aproduct item purchased by the user, from many product items bypresenting product items of other categories included in a product itemcombination that is thought to form an excellent combination with thepurchased product when product items of several categories included inthe product item combination that is configured by a plurality ofcategories of product items are purchased. In descriptions below, thetotal products (products of categories such as a blazer, a coat, ashirt, and pants) of garments are assumed to be the product items, theproduct items actually sold by the virtual store are assumed to be theproduct items, and each product item that is combined in an image mediumto be described later is described as a combined product item.

In this figure, the product search supporting system is configured by aproduct search supporting device 100 and one or a plurality of userterminals 200. The product search supporting device 100 has a controlportion 21, a similar-item searching portion 22, a combination searchingportion 23, a product searching portion 24, a transmission portion 25, aproduct item database 26, a product item image database 27, acombination information database 28, a combined product imageinformation database 29, and a combined product image database 30. Theuser terminal 200, for example, is a personal computer that is installedat each user's home and includes an internet browser. The product searchsupporting device 100 and the user terminal 200 are connected togetherthrough an information communication network I that includes theInternet.

The product item database 26 has memory areas configured as a pluralityof tables for types of the product items.

In other words, the product item database 26, as represented in FIG. 8,has a memory area configured as a table in which the attributeinformation of the product items that are to be sold is stored foridentification information A1 to An of the product item of a shirt asCategory 1.

In addition, similarly, the product item database 26, as represented inFIG. 9, has a memory area configured as a table in which the attributeinformation of corresponding product items for the identificationinformation B1 to Bn of the product item of pants as Category 2 isstored.

In addition, similarly, the product item database 26, as represented inFIG. 10, has a memory area configured as a table in which the attributeinformation of corresponding product items for the identificationinformation C1 to Cn of the product item of a jacket as Category 3 isstored.

Furthermore, the product item database 26, although not represented inthe figure, has memory areas in which a plurality of tables for shoes, ablouse, a bag, or the like as categories of the garments other than theabove-described categories is stored.

In addition, in each table of the product item database 26, asimilar-product item field is set up as one type of the attributeinformation. Thus, the product items of the combined product imageinformation database 29 that are the same as or similar to the productitem are written so as to be associated as a group of product items thatis similar to each purchase product item, as represented in FIG. 11(Category 1—shirt), FIG. 12 (Category 2—pants), and FIG. 13 (Category3—Jacket).

Here, the attribute information includes the identification informationof combined product items (product items included in the combinedproduct image information database 29 to be described later) that can beassociated to be the same or similar to each other, the characteristicsdata of an image that is extracted from the image data of the productitem, sales information such as the price, the brand, and the like ofthe product item, and the like. The types of the characteristics data ofthe image and a method of acquiring the characteristics data will laterbe described.

In the product item image database 27, image data for each product itemof each category that is stored in the product item database 26 isstored in association with the identification information of eachproduct item.

In the combination information database 28, garment products that areworn by a model or anyone else as a combination in a fashion magazine, afashion catalog, and an image medium on the Internet, that is, theproduct items that are worn as the combination are set as combinedproduct items, and each combination of the correspondence relationshipbetween a combined product item and a combined product item combinedthereto is stored in association with the identification information.For example, when a model wears the product items of a shirt havingidentification information a1-1, pants having identification informationb1-2, and a jacket having identification information c1-7 in a fashionjournal, as represented in the first row in the combination informationdatabase 28 represented in FIG. 14, a combined product item configuredby the identification information a1-1, the identification informationb1-2, and the identification information c1-7 is stored as a set inassociation with one another.

In this combination information, a combination of product items that aremanufactured as a combination by a fashion designer or a combination ofthe product items, which are manufactured by a fashion designer,combined by a fashion coordinator is the combination of the combinedproduct items. Accordingly, the combined product items form an elegantcombination. Thus, when such a combination is worn, there is a highpossibility that a person who sees the fashion accepts the combinationnaturally without having a feeling of oddity.

In the combined product image information database 29, combined productitems of garments that are acquired from fashion journals, fashioncatalogs, and image media (a material or a device that presents an imagesuch as a photo or an illustration of a journal) on the internet such asdesigner's correction information are accumulated for each category.Here, the combined product image information database 29, for example,is configured by tables having the configurations represented in FIGS.15, 16, and 17.

FIG. 15 represents a table in which shirts as Category 1 are accumulatedas the combined product items. FIG. 16 represents a table in which pantsas Category 2 are accumulated as the combined product items. FIG. 17represents a table in which jackets as Category 3 are accumulated as thecombined product items. In the combined product image informationdatabase 29, identification information is assigned to the combinedproduct item, and the attribute information of a corresponding combinedproduct item is stored for the identification information. Thisattribute information, for example, includes the identificationinformation of a similar product item, characteristics data of an imagethat is extracted from the image data of the combined product item,sales information such as the price and the brand of the combinedproduct item, and the like. The types of the characteristics data and amethod of acquiring the characteristics data will later be described.

In addition, in each table of the combined product image informationdatabase 29, a similar-product item field is set up as one type of theattribute information. Thus, the product items, included in the productitem database 26, that are the same as or similar to the combinedproduct item are written so as to be associated as represented in FIG.15 (Category 1—shirt), FIG. 16 (Category 2—pants), and FIG. 17 (Category3—Jacket).

In the table of the combined product image information database 29 thatis represented in FIG. 15, the identification information a1-1 to a1-kis stored as the combined product items that are the same as or similarto the product item A1, the identification information a2-1 to a2-k isstored as the combined product items that are the same as or similar tothe product item A2, and the identification information an-1 to an-m issequentially stored as the combined product items that are the same asor similar to the product item An.

Similarly, in the table of the combined product image informationdatabase 29 that is represented in FIG. 16, the identificationinformation b1-1 to b1-m is stored as the combined product items thatare the same as or similar to the product item B1, the identificationinformation b2-1 to b2-r is stored as the combined product items thatare the same as or similar to the product item B2, and theidentification information bn-1 to bn-m is sequentially stored as thecombined product items that are the same as or similar to the productitem Bn.

Similarly, in the table of the combined product image informationdatabase 29 that is represented in FIG. 17, the identificationinformation c1-1 to c1-q is stored as the combined product items thatare the same as or similar to the product item C1, the identificationinformation c2-1 to c2-k is stored as the combined product items thatare the same as or similar to the product item B2, and theidentification information cn-1 to cn-m is sequentially stored as thecombined product items that are the same as or similar to the productitem Cn.

In the combined product image database 30, the image data for everycombined product item of each category that is stored in the combinedproduct image information database 29 is stored in association with theidentification information of each combined product item.

In addition, the product item database 26, the product item imagedatabase 27, the combination information database 28, the combinedproduct image information database 29, and the combined product imagedatabase 30 that have been described above may be classified based onthe gender, the age, the trend of fashions (for example, a casual style,a conservative style, a brother style, an adult style, a street style, aUrahara (Harajuku) style, and a Mod style for men, and a girl style, asister style, a conservative style, a teen style, a celebrity style, andthe like for women) or the like.

When the databases are classified as above, the gender, the age (10's,20's, 30's, and the like), the fashion style, and the like must beacquired from the user as classification information.

The control portion 21 starting to operate as a virtual store managed bythe control portion 21 is accessed from the user terminal 200 andtransmits a process program (operates in accordance with a browser orthe like of the user terminal) for acquiring information on theclassification of the gender, the age, the fashion style, and the likeor performing a display and selection operation for a combination itemto be described later to the user terminal 200. Here, the controlportion 21 and the user terminal 200 perform data transmission and datareception through the information communication network I and thetransmission portion 25. The user terminal 200 starts theabove-described process program in an internal browser and performs datatransmission and data reception for the product search supporting device100 in the process for displaying an image or selecting a product itemor a combined product item.

In addition, when receiving reply data for the screen data from the userterminal 200, the control portion 21 selects one product item table froma plurality of product item databases 26 corresponding to the gender,the age, and the fashion style and transmits the category informationthat indicates the category of the combined product item, for example,text information or image information of a shirt, pants, a jacket,shoes, or the like to the user terminal 200 through the transmissionportion 25 and the information communication network I.

In addition, when receiving the type of the category selected by theuser which is transmitted from the user terminal 200, for example, thecategory information representing a shirt, the control portion 21selects a table represented in FIG. 8 for the category of the shirt(Category 1) as the product item database 26, reads in theidentification information A1 to An of the product items included inthis table, and reads in the image data of the product items having theidentification numbers A1 to An from the product item image database 27.Then, the control portion 21 converts the image data into image data ofa thumbnail image (reducing the number of bits of the image data) andtransmits the converted image data to the user terminal 200 with theidentification number of each corresponding product item added thereto.

In addition, when the identification number of the product item that isselected so as to be purchased from the thumbnail image by the user isinput, the control portion 21 performs an order receiving process forthe order and transmits the input identification number to thesimilar-item searching portion 22. Here, the order receiving processincludes billing charges corresponding to the price that is written inthe attribute information, checking stock of the ordered product item, adelivery procedure of the product item to the address input by the user,and the like for the user terminal 200.

The similar-item searching portion 22 searches for a plurality ofcombined product items having the similar-product item fields, in whichthe identification number that coincides with the identification numberof the input product item is written, from the table (Category 1) of thecombined product image information database 29, extracts combinedproduct items that are the same as or similar to the above-describedproduct item, and outputs the extracted combined product items to thecombination searching portion 23.

The combination search portion 23 searches for the product items of adifferent category, for example, the product items of the pants(Category 2) corresponding to the similar product items having theidentification number that is input from the similar-item searchingportion 22 from the combination information database 28, extractscorresponding product items, and outputs the identification informationof the combination to the control portion 21 as a second combinedproduct item.

In addition, the control portion 21 reads out the image datacorresponding to the identification information of a plurality ofcombinations of the identification information of the similar productitem input from the combination searching portion 23 and theidentification information of the second combined product item from thecombined product image database 30 in correspondence with theidentification information, transmits the read-out image data to theuser terminal 200, and outputs received identification number to theproduct searching portion 24 at a time when the identificationinformation of the combined product item of the combination selected bythe user is received.

The product searching portion 24 searches for the product item, which isactually sold, corresponding to the received identification informationof the second combined product item from the product item database 26,sets the found product item as a recommended product item of a differentcategory that is combined with the product item purchased by the user,reads out the image data of the recommended product item from theproduct item image database 27 by using the identification information,and transmits the read-out image data to the user terminal 200.

In addition, the product search portion 24 may be configured todetermine whether the amount of the product item set as the recommendedproduct is in the set range (for example, within 0.5 to 2 times of theamount of the purchased combined product) by comparing the amount withthe amount of the product item purchased by the user, transmit therecommended product to the user terminal 200 in cases where the amountof the recommended item is equal to or smaller than the purchased amountand to not transmit the recommended product for a case where the amountof the recommended item is out of the range.

Next, the operation of the product search supporting system according tothis embodiment will be described with reference to FIGS. 7 and 18. FIG.18 is a flowchart representing an example of the operation of theproduct search supporting system represented in FIG. 7. Hereinafter, aproduct input by the user is described as a product item, products thatare actually sold in the virtual store is described as product items(the product item database 26), and a product extracted from the imagemedium that is used at the time when a combination of products isselected is described as a combined product item (the combined productimage information database 29).

When the user terminal 200 accesses the virtual store that is managed bythe product search supporting device 100 through the informationcommunication network I in accordance with the user's operation, thecontrol portion 21 transmits a process program for displayinginformation for acquiring information on the gender, the age, and thefashion style and the combined item to be described later or performingan selection operation for a displayed image to the user terminal 200.Within the user terminal 200, an image display processing portion and aselection processing portion are included within the internet browser bythe above-described process program, and the user terminal 200 displaysthe image data transmitted from the product search supporting device 100and performs an editing process for the image data and a selectionprocess for the image data.

Then, when receiving reply data (for example, data selected from aplurality of selection options displayed on the screen) for the gender,the age, and the fashion style that are input to the screen data fromthe user terminal 200, the control portion 21 selects a table out of aplurality of tables of the product item database 26 corresponding to thegender, the age, and the fashion style and transmits the categoryinformation that indicates the category of the product item, forexample, text information or image information of a shirt, pants, ajacket, shoes, or the like to the user terminal 200 through thetransmission portion 25 and the information communication network I.

When the text information or the image information is received, the userterminal 200 displays the category (for example, a shirt, pants, ajacket, or the like) on the basis of the text information or the imageinformation in a display portion in accordance with the above-describedprocess program and performs display (for example, “Please select” orthe like) for urging the user to select any one for purchase.

When the user selects several categories, the user terminal 200transmits the category information that indicates the categoriesselected by the user, for example, the category information indicatingthe shirt to the product search supporting device 100 (Step F1).

In other words, when receiving the category information of a productitem desired to be purchased by the user, the control portion 21 selectsthe table, which is represented in FIG. 8, having the shirt as thecategory (Category 1) out of the tables of the product item database 26represented in FIGS. 8 to 10 and reads out all the identificationinformation A1 to An of the product items included in this table. Then,the control portion 21 reads out the image data of the product itemshaving identification numbers A1 to An from the product item imagedatabase 27 and transmits the image data as the image data of the thumbnail image to the user terminal 200 with the identification numbers ofthe product items corresponding to the thumb nail images added thereto.

When receiving the image data of the thumbnail images, the user terminal200 displays the thumbnail images of the product items having theidentification numbers A1 to An in the display portion (F2).

Then, when the user selects any one of the displayed thumbnail images,the user terminal 200 transmits the identification information of theselected thumbnail image to the product search supporting device 100.

When receiving the identification information of the selected thumbnailimage, the control portion 21 performs an order receiving process forthe order for the product item of the identification number, that is, ashirt as the product item selected as a purchase target from thethumbnail images by the user and transmits the received identificationnumber to the similar-item searching portion 22.

Then, when receiving, for example, the identification number A1 of theproduct item purchased by the user, the similar-item searching portion22 extracts a candidate group that is formed by one or a plurality ofcombined product items that coincide with or is similar to theidentification number A1 from the tables of the combined product imageinformation database 29 represented in FIGS. 15 to 17 (Step F3).

Here, for example, when receiving the identification number A1 of theproduct item from the combined product image information database 29,the similar-item searching portion 22 extracts the combined productitems having the identification numbers a1-1 to a1-q in which theidentification number A1 is written in the similar-product item field ofthe attribute data as a candidate group of a similar combined productitem and outputs the combined product item to the combination searchingportion 23.

The combination searching portion 23 searches for combined product itemsof a different category, for example, combined product items of pantscorresponding to the combined product items of the identificationnumbers of the candidate group that are input from the similar-itemsearching portion 22 from the combination information database 28represented in FIG. 14 and extracts the corresponding combined productitems (for example, b1-2, b2-3, b1-4, . . . corresponding to a1-1, a1-2,a1-3, . . . represented in FIG. 14) of the pants, and outputs theidentification information of the combination to the control portion 21as the second combined product item (Step F4). Here, the categoryrecommended as the combined product item combined with the product itempurchased by the user may be set in advance in correspondence with thepurchased product or may be initially selected by the user from aplurality of categories.

Then, the control portion 21 reads out image data corresponding to aplurality (corresponding to the number of combined product itemscorresponding to the candidate group) of combinations of theidentification information of the combined product item input from thecombination searching portion 23 and the identification information ofthe second combined product item from the combined product imagedatabase 30 in association with the identification information andtransmits the read-out image data of each product item to the userterminal 200 with corresponding identification information addedthereto.

When the plurality of combinations of the identification information(corresponding to the shirt) of the combined product item and theidentification information (corresponding to the pants) of the secondcombined product item and the image data corresponding thereto arereceived, the user terminal 200 displays the image data of eachcombination in the display portion (Step F5).

In addition, a plurality of three-dimensional human images correspondingto each body type, which are created by CG (computer graphics), isdisplayed in the end portion of the display screen, and by selecting anyone of the plurality of the human images, the user terminal 200 displaysthe shirt and the pants of the combined product item so as to beoverlapped with the human image. In addition, it may be configured thatthe user photographs his or her face by using a mounted web camera, andthe user terminal 200 performs an image processing for displaying theface area, which is selected by the user, so as to be overlapped withthe face portion of the CG human face.

Next, when the user selects any one of the plurality of combinationsdisplayed in the display portion (Step F6); for example, when the userselects a combination of the identification information a1-1 and theidentification information b1-2, the user terminal 200 transmits theidentification information b1-2 of the second combined product item(corresponding to the pants) of the combination selected by the user tothe product search supporting device 100.

When receiving the identification information b1-2 of theabove-described second combined product item through the control portion21, the product searching portion 24 searches the table, which isrepresented in FIG. 9, of the product item database 26 corresponding tothe category of the pants (Step F7) and extracts the product item havingthe identification information B1 corresponding to the identificationinformation b1-2 (Step F8).

Then, the control portion 21 searches for the image data correspondingto the identification information B1 of the combined product itemextracted by the product searching portion 24 from the product itemimage database 27 and reads out the found image data.

In addition, the control portion 21 searches for and reads out the imagedata corresponding to the identification information A1 purchased by theuser from the product item image database 27 and transmits the read-outimage data together with the image data of the product item having theidentification information B1 to the user terminal 200.

When receiving the image data of the product item, the user terminal 200displays combined image data in the display portion (Step F9).

At this moment, same as in the above-described Step F5, a plurality ofthree-dimensional human images for each body type, which is created bythe CG, is displayed in the end portion of the display screen. Thus, byselecting any one of them, the user terminal 200 displays the shirt(identification information A1) and the pants (the identificationinformation B1) of the product items so as to be overlapped with theabove-described human image. In addition, the user terminal 200 may beconfigured so as to photograph the user's face by using a mounted webcamera, and displays the area of a face selected by the user so as to beoverlapped with the face portion of the above-described human image.

Then, when the user selects the purchase or non purchase of therecommended product out of the product items having the identificationinformation B1 from the option displayed on the display screen (StepF10), the user terminal 200 transmits the identification information A1and the identification information B1 together with information thatindicates the purchase or non-purchase of the recommended product to theproduct search supporting device 100.

When receiving the identification information A1 and the identificationinformation B1 together with the information that indicates the purchaseor non purchase as input, in cases where the purchase is selected, thecontrol portion 21 performs an order receiving process, same as for thecase of the product item having the identification information A1.

In addition, a history database 31 denoted by a broken line representedin FIG. 7 may be provided in the product search supporting device 100.

The history database 31 is configured by a user table configured as atable represented in FIG. 19 and a purchase history table for each userwhich is configured as a table represented in FIG. 20.

The user table represented in FIG. 19 is configured such that useridentification information that is assigned to each user who has beenregistered as a member or has purchased a product item and is used foridentifying each user and at least the user's name and the user's mailaddress corresponding to the user identification number can beassociated with each other.

In addition, the purchase history table represented in FIG. 20 is set upfor each user. In the purchase history table, in association with theaccess date and time at which the virtual store is accessed,identification information of the product item that has been purchasedat the access date and time, identification information of a productitem that has been searched for and not purchased, and identificationinformation of a combined product item that has been recommended for thepurchased product item and not been purchased, which are identified bythe above-described user identification information, are stored ashistory. When there is no product that has been purchased “−” is stored.

For example, when a user accesses the virtual store from the userterminal 200 and performs membership registration by inputting his orher name and mail address in a membership registration page, the controlportion 21 assigns identification information to the user andadditionally registers the user in the user table.

Then, when the user searches for a product item of a category selectedby the user and purchases the product item, the control portion 21stores the purchased product item in the area for the identificationinformation of the purchased product items. On the other hand, when theuser searches for a product item but does not purchase the product item,the control portion 21 stores the identification information of theproduct item that has been searched for in the area for theidentification information of the product items that have been searchedfor but have not been purchased.

In addition, when the user purchases a product item as a recommendedproduct that is combined with the above-described purchased productitem, the control portion 21 stores the purchased product item in thearea of the above-described purchase history table for theidentification information of the purchased product items. On the otherhand, when the user does not purchase the product item as therecommended product item, the control portion 21 stores theidentification information in the area for the identificationinformation of the product items that have been recommended but not beenpurchased.

Then, when the user accesses the virtual store again and searches for aproduct item, the control portion 21 selects the user's purchase historytable from the purchase history database 31 by using the useridentification information input by the user and searches the selectedpurchase history table by using the identification information of theproduct item.

At this moment, when the identification information of the product itemthat the user searches for is detected in the area for theidentification information of the purchased product items, the controlportion 21 transmits the product item to the user terminal 200 with thepurchased date and time of the same product item added thereto so as tobe displayed on the display screen, whereby notifying the user of thecombined product item that has already been purchased.

On the other hand, when the identification information of the productitem, which is searched for, is detected in the area for theidentification information of the product items that have been searchedfor but not been purchased, the control portion 21 notifies the user ofinformation that stimulates the purchase desire such as “Is this productitem a product item that was also searched for in the past and is thetype you like?” by transmitting the product item with the search dateand time for the same product item added thereto to the user terminal200 so as to be displayed on the display screen.

On the other hand, when the identification information of the productitem, which is searched for, is detected in the area for theidentification information of the product items that have beenrecommended but not been purchased, the control portion 21 notifies theuser of information that stimulates the purchase desire such as “Thisproduct item is an excellent combination with the product item purchasedin the past” by transmitting the product item with the search date andtime for the same product item added thereto to the user terminal 200 soas to be displayed on the display screen.

Next, the characteristics data will be described. For example, thecharacteristics data is acquired by performing a two-dimensional Fouriertransform for the pattern of the fabric of the garment for each colorspace of R (red), G (green), and B (blue) as represented in FIGS. 8 to10 and FIGS. 15 to 17. The control portion 21 creates element data RD,GD, and BD of the characteristics data by performing the two-dimensionalFourier transform. At this moment, when the product item or the combinedproduct item is a shirt or a jacket, the horizontal width or theshoulder width of the clothes is used as a reference value of the lengthfor the two-dimensional Fourier transform, so that data is matched inthe process for each product item and the combined product item.

On the other hand, when the product item or the combined product item ispants, the horizontal width of the waist portion is used as a referencevalue of the length for the two-dimensional Fourier transform, so thatdata is matched in the process for each product item and the combinedproduct item.

In other words, since the similarity is checked by using thecharacteristics data, in order to determine the size of the pattern andthe like as the attributes, the sizes of the total portions need to benormalized by using the measurements of several positions of theaccessory, so that the results of the two-dimensional Fourier transformfor the combined products included in the same category are matched.When the image data for which the two-dimensional Fourier transform isperformed is to be photographed, a shirt, a jacket, pants, and the likeare photographed by using a digital camera or the like after beingflattened on a flat floor.

In addition, a shape as the element data of the characteristics data,for example, is a ratio of the length of the sleeve to the shoulderwidth for a shirt or a jacket and is a ratio of the thigh width to thebottom width for pants.

Next, the texture is acquired by enlarging a fabric portion having thelargest area and performing a two-dimensional Fourier transform for theenlarged fabric portion. At this moment, data for the two-dimensionalFourier transform that can be acquired from the image data for differentcombination items is matched by fixing the enlargement ratio at aconstant value.

As described above, the store personnel of the virtual store, by usingthe control portion 21, collects the characteristics data from the imagedata of the product items that are sold and collects the attribute datafrom the image data of combined product items that is collected fromimage media such as fashion catalogues or the Internet.

Then, for the combined product items having similarity, clustering ofthe combined product items collected from the image media is performedby using the characteristics data of the product items that are actuallysold as the center data of the cluster. Here, the control portion 21,for example, calculates a distance between a comparative characteristicsvector that is configured by characteristics data of the image data ofeach combined product item collected from the image media input by thestore personnel and a reference characteristics vector that isconfigured by the characteristics data of the image data of each productitem that is actually sold. Then, the control portion 21 performs aprocess in which a combined product item having the characteristicsvector that has a distance from each product item shorter than that ofthe reference characteristics vector of other combined product items isregarded as a cluster having the similarity for the product itemregarded to be close from the combined product item, whereby generatingthe correspondence relationships between the combined product itemsincluded in the tables of FIGS. 15, 16, and 17 and the product itemplaced in the similar product item field. Similarly, the correspondencerelationships between the product items included in the tables of FIGS.8, 9, and 10 and the combined product items placed in the similarproduct item field are generated. In other words, the identificationinformation written in the similar product item field of the table ofFIGS. 8, 9, and 10 is the identification information of combined productitems having shorter distances, that is, having similarity relative toother product items that are sold for the product item, which is sold,having the corresponding identification information.

Here, the control portion 21, for example, may be configured to storethe combined product items so as to be aligned in the similar productitem fields of each figure of FIGS. 8 9, and 10 in the order of shorterdistances, that is, in the order of higher similarities. Accordingly,when a combined product item that is the most similar to the productitem is to be selected, the similar-item searching portion 22 canextract the most similar combined product item or combined product itemsup to the combined product item that has the h-th highest similarityfrom the highest similarity side (h is set in advance) from the combinedproduct items, which are collected from the image media, more easilythan those of FIGS. 15 to 17.

In addition, instead of performing clustering in advance by comparingthe reference characteristics vector of each product item with thecharacteristics data of a combined product item, which has beencollected from the image medium, having a comparative characteristicsvector that has a short distance and arranging the fields for writingthe identification information of the similar product items, asdescribed above, the similarity may be configured to be calculated eachtime when the similar combined product item is searched for.

For example, in searching for the combined product item, which has beenacquired from the image medium, similar to the product item in Step F3,a configuration in which the similar-item searching portion 22calculates the similarity (the similarity is high as the distance isshorter) based on the distance between the above-described referencecharacteristics vector and the comparative characteristics vector, andthe combined product item, which has been collected from the imagemedium, having similarity to the product item is extracted may be used.

At this moment, similarly, in searching for the product item that issimilar to the combined product item acquired from the image medium inStep F7, a configuration in which the product searching portion 24calculates the similarity based on the distance between theabove-described reference characteristics vector and the comparativecharacteristics vector, and the product item having similarity to thecombined product item collected from the image medium is extracted maybe used.

Furthermore, it may be configured that the similar-item searchingportion 22 and the product searching portion 24 calculate the distancebetween the product item and the combined product item collected fromthe image medium and extracts the product item having the highestsimilarity or product items up to a product item having the h-th highestsimilarity from the highest similarity side.

As described above, according to this embodiment, in purchasing combinedproducts, when a product that matches well for a case where the productis combined with the product that is to be purchased or has beenpurchased by the user is extracted from a product group that is sold anda matching product is selected from many products of many storesarranged in the virtual store on the Internet for recommending theextracted product to the user, the user can easily select the combinedproducts without worrying about the match of the combined products.

Third Embodiment

In the second embodiment, the product search supporting device 100transmits the product items to the user terminal 200 by using thethumbnail images for being selected by the user on the screen of theuser terminal 200.

According to the third embodiment, it may be configured that image dataof a garment desired to be purchased which has been read in from theimage medium by using a scanner or the like or been downloaded from theInternet after the user accesses the virtual store is input to the userterminal 200 as the image data of the product item, the image data istransmitted to the product search supporting device 100 by the userterminal 200, and the product search supporting device 100 extracts aproduct item that is similar to the combined product item of the imagedata. The operation after extraction of the product item is the same asthat after the user's selection of the product item from the thumbnailimages in the second embodiment.

The configuration of the product search supporting device 100 accordingto the third embodiment is the same as that of the second embodiment.Hereinafter, only operations that are different from those of the secondembodiment will be described.

When the user accesses the virtual store by using the user terminal 200,the product search supporting device 100 starts to operate, and thecontrol portion 21 transmits the image data of an input screen (optionsare clicked by using a mouse or the like for a search) used forinquiring of the user terminal 200 whether the product item is selectedfrom thumbnail images or the similar product item is extracted from theproduct item database 26 by using the image data of the garment that isinput by the user to the user terminal 200.

Then, when a reply signal indicating selecting by using the thumbnailimages is input from the user terminal 200 by user's determination onselecting a product to be purchased from the thumbnail images, thecontrol portion 21, same as in the second embodiment, transmitsthumbnail images of a plurality of product items to the user terminal200 with identification information of the product items added to eachimage data. Thereafter, the process is the same as that of the secondembodiment after the user selects any of the thumbnail images.

On the other hand, when the user selects to extract a similar productitem from the product item database 26 by using the image data input bythe user, the user terminal 200 transmits a reply signal that representsselecting of the product item by using the image data input by the userto the product search supporting device 100. Accordingly, the controlportion 21 transmits information on the input screen from which theimage data is input to the user terminal 200.

The user allows the image data (a paper image medium such as a fashionjournal or a fashion catalogue) of a garment desired to be purchased tobe read in by the user terminal 200 by using a scanner, or the userphotographs by using a digital camera and allows the user terminal 200to read in the photographed image data or the image data acquiredthrough the Internet.

When the image data is read in, the image data is displayed in the imagedata display area of the input screen of the user terminal 200 asrepresented in FIG. 21. Then, the user selects the garment portion ofthe image data desired to be purchased, for example, a shirt portiondesired to be extracted by using a broken line H1 and selects a portion(a portion near outer periphery portion of the shirt) other than theshirt portion desired to be extracted selected by a broken line 2,selects the category name by using a combo box, and then clicks on thetransmission button located on the display screen. Accordingly, the userterminal 200 detects that a process for transmitting the image data isrequested and transmits the image data of the garment drawn by thebroken line H1 and the broken line H2, the image data of the garmentthat is not drawn, and category information representing the categoryname of the selected garment to the product search supporting device100.

Then, when receiving the image data of the garment drawn by the brokenline H1 and the broken line H2 and the image data not drawn by a brokenline, the control portion 21 performs a process for dividing the imagedata of the garment into the shirt portion and the other portion byusing a Graph-Cut method, whereby extracting the shirt portion.

In other words, the control portion 21 performs division by calculatingthe boundary between the area having a pixel value that is the same asthe pixel located on the broken line H1 that is drawn in the shirtportion and an area having the same pixel value as that of the pixel onthe broken line H2 drawn other than the shirt portion as a position inwhich the error of the error function on the basis of the gradientbecomes the minimum.

Here, when the shirt portion is extracted, the control portion 21performs the two-dimensional Fourier transform and extracts the shapedata as an element of the characteristics data and outputs the shapedata as the detection target characteristics data, as described above.

Then the control portion 21 selects a table corresponding to thecategory of the extracted garment, for example, the shirt tableaccording to this embodiment from the product item database 26.

After selecting the shirt table, the control portion 21 calculates adistance between the reference characteristics vector that is configuredby characteristics data of the product items of the table and the targetcharacteristics vector that is configured by the above-describeddetection target characteristics data and extracts the product itemsthat have characteristics data similar to the detection targetcharacteristics data from the shirt table up to the product item havingthe h-th highest similarity; for example, product items having up to thefifth highest similarity. Here, the control portion 21 cannot easilyextract the texture as the characteristics data from the image data thathas been transmitted from the user. Thus, when the characteristicsvector is to be generated from the characteristics data, the distance ofboth characteristics vectors to be compared with each other iscalculated by excluding the texture from the elements of the vector.

Next, the control portion 21 reads out the image data of each productitem from the product item image database 27 based on the identificationinformation of the highest five product items and transmits the imagedata to the user terminal 200 as thumbnail images in association withthe identification information. The process thereafter is the same asthat after transmission of the thumbnail images in Step F2 of theflowchart according to the second embodiment represented in FIG. 18.

Fourth Embodiment

In the third embodiment, the product item identical to the image data ofthe product item input by the user or the same product item is searchedfor from the product item database 26 in which product items that aresold are stored and the found product item is presented to the user, anda process for recommending the product item of a different category thatis combined with the found product item is performed.

According to the fourth embodiment, an operation for searching for aproduct item that is combined with the garment owned by the user inadvance is performed.

The user photographs the garment owned by him or her, for example, ashirt by using the digital camera and allows the user terminal 200 toread in the image data of the shirt by using the digital camera.

Then, when the user accesses the virtual store by using the userterminal 200, the product search supporting device 100 starts tooperate. When the user accesses the virtual store by using the userterminal 200, the product search supporting device 100 starts tooperate, and the control portion 21 transmits the image data of an inputscreen (options are clicked by using a mouse or the like for a search)used for inquiring the user terminal 200 whether the product item isselected from thumbnail images, the similar product item is extractedfrom the product item database 26 by using the image data of the garmentthat is input by the user, or a combined product item is recommended tothe garment of the image data input by the user to the user terminal200.

Then, when a response signal indicating selecting by using the thumbnailimages is input from the user terminal 200 by user's determination onselecting a product to be purchased from the thumbnail images, thecontrol portion 21, same as in the second embodiment, transmitsthumbnail images of a plurality of product items to the user terminal200 with identification information of the product items added to eachimage data. Thereafter, after the user selects any of the thumbnailimages, the process is the same as that of the second embodiment.

On the other hand, when the user selects to extract a similar productitem from the product item database 26 by using the image data input bythe user, the user terminal 200 transmits a reply signal that representsselecting of the product item by using the image data input by the userto the product search supporting device 100. The process thereafter isthe same as that of the third embodiment.

On the other hand, when the user selects recommendation of a productitem of a different category that is combined with the garment of theimage data input by the user, the user terminal 200 transmits a replysignal indicating recommendation of the product item combined with thegarment of the image data input by the user to the product searchsupporting device 100.

Hereinafter, the process for recommending a product item combined withthe garment of the image data input by the user according to the fourthembodiment will be described. The configuration of the product searchsupporting device 100 according to the fourth embodiment is the same asthat according to the second embodiment. Hereinafter, only operationsthat are different from those of the first and third embodiments will bedescribed.

As described above, when the user selects recommendation of the productitem combined with the garment of the image data input by the user, theuser terminal 200 transmits a reply signal that indicates recommendationof the product item combined with the garment of the image data input bythe user to the product search supporting device 100. Accordingly, thecontrol portion 21 transmits the information on the input screen fromwhich the image data is input to the user terminal 200.

The user photographs the image data of the garment (for example, ashirt) owned by him or her for which the combined product item isdesired to be recommended by using an imaging device such as a digitalcamera and allows the image data acquired by photographing the garmentto be read in by the user terminal 200.

When the image data read in by the user terminal 200 is displayed, theuser inputs the category of the garment of the image data displayed inthe image data display area by selecting the category from the combobox.

Then, when the user selects the transmission button located on the inputscreen by using a pointing device such as a mouse, the user terminal 200detects that a process for transmitting the image data has beenrequested and transmits the image data displayed in the image datadisplay area to the product search supporting device 100 together withthe category information that represents the above-described category.

When the image data is input, the control portion 21 performs thetwo-dimensional Fourier transform for the garment of the image data, forexample, the image data of the shirt and extracts the shape data as anelement of the characteristics data and outputs the shape data as thedetection target characteristics data to the similar-item searchingportion 22 together with the category information.

Then, the similar-item searching portion 22 selects a tablecorresponding to the category of the extracted garment, for example, ashirt table from the combined product image information database 29.

After, selecting the shirt table, the similar-item searching portion 22calculates a distance between the reference characteristics vector thatis configured by the characteristics data of the combined product itemsincluded in the table and the target characteristics vector that isconfigured by the detection target characteristics data and extracts theproduct items that have characteristics data similar to the detectiontarget characteristics data from the shirt table up to the product itemhaving the h-th highest similarity; for example, combined product itemshaving up to the fifth highest similarity. Here, the control portion 21cannot easily extract the texture as the characteristics data from theimage data that has been transmitted from the user. Thus, when thecharacteristics vector is to be generated from the characteristics data,the reference characteristics vector and the detection targetcharacteristics vector are generated by excluding the texture from theelements of the vectors.

Then, the similar-item searching portion 22 output the combined productitems having the detecting similarity up to the fifth highest combinedproduct item; for example, identification information corresponding tofive shirts to the combination searching portion 23.

When the identification information of the combined product item isinput, the combination searching portion 23 reads out the identificationinformation of a different category that is stored in association withthe identification information of five input combined product items; forexample, the identification information of the combined items of pantsfrom the combination information database 28.

Next, the combination searching portion 23 transmits combinations of theidentification information of five combinations between the shirt andthe pants to the control portion 21. The process performed thereafter isidentical to the process performed from Step F5 in the flowchart of FIG.18.

Fifth Embodiment

In the combined product image information database 29, the attributedata of the image data of the old garment that is collected from oldfashion journals or fashion catalogues published in the past, forexample, 10 years ago or 20 years ago may be stored in association withthe identification information.

In addition, in the combined product image database 30, the image dataof the old garment is stored in association with the above-describedidentification signal.

Then, in association with the combination of the combined product itempublished in fashion journals or fashion catalogues in the past, a tableof a combination of the combined product items of different categoriesthat is represented in FIG. 14 is generated in the combinationinformation database 28.

As described above, by configuring the combined product imageinformation database, the combined product image database 30, and thecombination information database 28, in cases where a new product itemthat is similar to the old design in the past, it becomes possibleeasily to extract combined product items for a new combined product itemthat is similar to the old design of the past from the product item thatis currently sold by using the combination of the old design of thepast.

In this embodiment, a garment has been described as an example of theproduct item. However, this embodiment can easily be applied to theentire combined products acquired by combining a plurality of productsof different types (categories), for example, combined furniture,combined electric appliances, or the like.

In addition, in each of the above-described embodiments, the process forsupporting a product search may be performed by recording a program forimplementing the functions of each portion of the search servers 1 and 1a and the product search supporting device 100 in a computer-readablerecording medium, reading the program recorded in this recording mediuminto the computer system, and executing the program. The “computersystem” described here includes an OS and hardware such as peripheraldevices. In addition, the “computer system” includes a WWW system thathas a home-page providing environment (or display environment). Inaddition, the “computer-readable recording medium” means a portablemedium such as a flexible disk, an optical magnetic disc, a ROM, aCD-ROM or the like and a memory device such as a hard disk that is builtin the computer system or the like. In addition, the “computer-readablerecording medium” includes a device such as a volatile memory (RAM) fora case where the program is transmitted through a network such as theInternet or a communication network such as a telephone line that storesthe program for a predetermined time.

Furthermore, the program may be transmitted from the computer system inwhich this program is stored in a memory device or the like to anothercomputer system through a transmission medium or a carrier waver in thetransmission medium. Here, the “transmission medium” for transmittingthe program means a medium that has a function for transmittinginformation including a network (communication network) such as theInternet or the communication circuit line (communication line) such asthe telephone line. In addition, the program may be used forimplementing a part of the above-described function. Furthermore, onethat can implement the above-described function by being combined with aprogram that is recorded in the computer system in advance, that is, aso-called a difference file (difference program) may be used.

The invention may be used very appropriately in a search supportingsystem that supports a user to search for a target and to select ordetermine a target to be searched for through the Internet andtechnology similar thereto, and a user's search for a target desired tobe searched for can be performed by using the image data of the target.

What is claimed is:
 1. A search supporting system comprising: a memorythat stores instructions; and a computer that is configured to executethe instructions stored in the memory to: receive an input image,extract characteristics of the input image, search a first database toacquire first product image data, which has image characteristicsinformation that is the same as or similar to that of the input image,the first product image data corresponding to a first product, search asecond database to acquire second product image data corresponding to asecond product, which is different from the first product and which canbe combined with the first product, combine images of the first productimage data and the second product image data to produce combined imagedata, and output (a) the first product image data, (b) information onthe second product and (c) the combined image data.
 2. The searchsupporting system according to claim 1, wherein the imagecharacteristics information includes at least one of the form, thecolor, the shape, and the texture.
 3. The search supporting systemaccording to claim 1, wherein the information on the second productincludes at least one of product image data, the name of the product, asales store that sells the product, and the price of the product.
 4. Thesearch supporting system according to claim 1, wherein the first productincludes at least one of goods, an electro-chemical product, furniture,a drawing, a building including a store or a company, a plant, ananimal, real estate, and a landscape.
 5. A method of supporting asearch, the method performed by a computer executing instructions storedin a memory and comprising: receiving an input image; extractingcharacteristics of the input image; searching a first database toacquire first product image data, which has image characteristicsinformation that is the same as or similar to that of the input image,the first product image data corresponding to a first product; searchinga second database to acquire second product image data corresponding toa second product, which is different from the first product and whichcan be combined with the first product; combining images of the firstproduct image data and the second product image data to produce combinedimage data; and outputting (a) the first product image data, (b)information on the second product and (c) the combined image data.
 6. Anon-transitory computer-readable storage medium that stores a searchsupporting program that causes a computer to perform the operations of:receiving an input image; extracting characteristics of the input image;searching a first database to acquire first product image data, whichhas image characteristics information that is the same as or similar tothat of the input image, the first product image data corresponding to afirst product; searching a second database to acquire second productimage data corresponding to a second product, which is different fromthe first product and which can be combined with the first product;combining images of the first product image data and the second productimage data to produce combined image data; and outputting (a) the firstproduct image data, (b) information on the second product and (c) thecombined image data.